Creating a config map to inject into a deployment:
kubectl create configmap webapp-config-map --from-literal=APP_COLOR=darkblue —from-literal=APP_SIZE=12 …
OR
Kubectl create configmap app-config --from-file=app_config.properties
OR
Kubectl create -f <below_yaml>
Creating configmap via yaml:
apiVersion: v1
kind: ConfigMap
metadata:
name: app-config
data:
APP_COLOR: blue
APP_MODE: prod
THEN replace the -env field with:
envFrom:
- configMapRef:
name: app-config
apiVersion: v1
kind: Secret
metadata:
name: app-secret
data:
DB_Host: awekm+12
DB_User: cdksfd==
DB_Password: cDGfSAa
IF GOING YAML: you need to encode via <echo -n “name” | base64> to convert into base 64 or else k8 won’t allow you to create the secret
Injecting env variables into pods
envFrom:
- secretRef:
name: app-secret
env:
- name: DB_Password
valueFrom:
secretKeyRef:
name: app-secret
key: DB_Password
volumes:
- name: app-secret-volume
secret:
secretName: app-secret
Under spec in pod yaml, can set the security for a user on the docker image for the pod:
spec:
securityContext:
runAsUser: 1000. #User id
OR for the container level:
spec:
containers:
securityContext:
runAsUser: 1000
capabilities:
add: [“MAC ADMIN”]
kubectl exec <podname> -- whoami
Provides an identity for processes that run in a pod
Create a service account:
Kubectl create serviceaccount <sa_name>
Request service account token:
kubectl create token <sa_name>
Viewing secret token:
kubectl describe secret <sa_tokens_found_in_describe_serviceaccount>
Attaching a custom serviceaccount on pod creation:
spec:
containers:
…
serviceAccountName: <sa_name>
Including specific requests/limits for container in a pod yaml:
spec:
containers:
-name:
image
…
resources:
requests:
memory: “1Gi”
cpu: 1
limits:
memory: “2Gi”
cpu: 2
Taint - a way in which we can label nodes for certain types of pods Toleration - a permission applied which pairs a pod with a node labeled with a taint
Creating a taint:
kubectl taint nodes <node_name> <key>=<value>:<taint-effect>
Identify a taint:
kubectl describe node <node_name> | grep Taint
Taint effect - what happens to pods that do not tolerate this taint? - NoSchedule: pod not scheduled on node - PreferNoSchedule: node prioritized last - NoExecute: new pods won’t be scheduled AND old pods without toleration will be booted
Giving a pod a toleration:
spec:
containers:
…
tolerations:
- key: “app”
operator: “Equal”
value: “blue”
effect: “NoSchedule”
Node selector- limiting a pod to a specific node
spec:
containers:
…
nodeSelector:
size: Large
*size=Large is a key/value label assigned to the node
kubectl label nodes <label_name> <label_key>=<label_value>
Node Affinity - ensure that pods are assigned to particular nodes by providing advanced boolean expressions
spec:
containers:
…
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: size
operator: NotIn
values:
- Small
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: size
operator: Exists
NOTE: Combinations of taints/tolerations AND node affinity sometimes need to be used to ensure that all pods are assigned to all anticipated nodes
It’s possible to place multiple containers into the same pods so that they can have access to the same network and storage.
Don’t have to establish volume sharing or services between pods to enable communication between.
yaml example:
apiVersion: v1
kind: Pod
metadata:
name: simple-webapp
labels:
name: simple-webapp
spec:
containers:
- name: simple-webapp
image: simple-webapp
ports:
- containerPort: 8080
- name: log-agent
image: log-agent
Three design patterns:
Init containers are special types of containers which: a. will only run once when the pod is first created b. a process that waits for an external service/database to be up before the actual application starts
Init containers must first run to completion before the real container hosting the application starts (pod will restart repeatedly if any fail occurs)
Multiple initContainers can be configured in one pod (will be run sequentially)
Example yaml:
apiVersion: v1
kind: Pod
metadata:
name: myapp-pod
labels:
name: myapp-pod
spec:
containers:
- name: myapp-container
image: busybox:1.28
commands: ['sh', '-c', 'echo The app is running! && sleep 3600']
initContainers:
- name: init-myservice
image: busybox
command: ['sh', '-c', 'git clone <target-repo> ;']
Pre-Reqs:
Pod Statuses:Pod Conditions (True/False): PodScheduled, Initialized, ContainersReady, Ready
Ready - pod is running and ready to accept user traffic, but this has different meanings in different contexts…
*** This can be problematic!! Especially if a service directs to a container that is “Ready” but not really ready.
Readiness probes: ways in which one can actually identify if a container is truly ready
Pod yaml examples:
web app: http test to see if the server responds (/api/ready)
containers:
- name:
...
readinessProbe:
httpGet:
path: /api/ready
port: 8080
database: TCP Test to see if a socket is listening (3306)
containers:
- name:
...
readinessProbe:
tcpSocket:
port: 3306
exec: create a command that will succesfully complete if the container is ready (cat /app/is_ready)
containers:
- name:
...
readinessProbe:
exec:
command:
- cat
- /app/is_ready
*** Back to the service example, an httpGet test can be used on each container to verify with the service that the pod is ready to accept traffic.
Liveness Probes: useful for when an application is “broken” but the container continues to run (ex: bug in code keeps app stuck in infinite loop)
Periodically checks the health of an application (this is what we get to define!)
JUST LIKE READINESS, but replace “readinessProbe” with “livenessProbe”
There does exist extra options!
livenessProbe:
httpGet:
path: /api/healthy
port: 8080
initialDelaySeconds: 10
periodSeconds: 5
failureThreshold: 8
Aside: You can run images on Docker and identify the logs via the following commands
docker run -d <image-name>
docker logs -f ecf
Note: -d=daemon, -f=live log trail
In the same way, we can pull the logs of a pod via a live stream with the following command:
kubectl logs -f <pod-name> <container-name-if-multiple>
Common open source monitoring software: Metrics Server, Prometheus, Elastic Stack, Datadog, dynatrace
Metrics Server: * Can have one metric server per k8 cluster * Only an in-memory monitoring solution (can’t see historical performance!)
kubelet - agent run on each node which is responsible for receiving
instructions from master node and running pods in nodes
- also
contains sub-component cAdvisor which is responsible for receiving pod
metrics and passing along to monitoring software via API
To enable the metrics server, run the following:
minikube: minikube addons enable metrics-server
other: git clone https://github.com/kubernetes-incubator/metrics-serve
kubectl create -f deploy/1.8+/
can then see monitoring vals via:
kubectl top node
kubectl top pod
labels: standard method to group and filter objects together based on criteria
selectors: aid in filtering labels
under “metadata” as many labels can be used as desired!
apiVersion: v1
kind: Pod
metadata:
name: simple-webapp
labels:
app: App1
function: Front-end
...
can then select specific labels via the –selector flag:
kubectl get pods --selector app=App1
getting all objects with a certain label:
kubectl get all --selector app=App1
getting objects with multiple label criteria:
kubectl get pods --selector env=prod,bu=finance,tier=frontend
matchLabels - ties the replicaset to the pods (can use multiple if you think there will be other pods with the same label but different functionality)
example:
apiVersion: apps/v1
kind: ReplicaSet
metadata:
name: simple-webapp
labels:
app: App1
function: Front-end
spec:
replicas: 3
selector:
matchLabels:
app: App1
template:
metadata:
labels:
app: App1
function: Front-end
spec:
containers:
- name: simple-webapp
image: simple-webapp
Annotations: used to record other details for information purposes
When a deployment is created, a new rollout is first created. A new rollout creates a new deployment revision.
Helps us keep track of changes from deployment to deployment
Tracking status of deployment:
kubectl rollout status deployment/myapp-deployment
History/revisions of a deployment:
kubectl rollout history deployment/myapp-deployment --revision=<optional-desired-revision-info>
Can either destroy instances, followed by recreating with newer… BAD!!! inaccessible to users for a time. INSTEAD destroy and upgrade a few at a time.
Update a deployment: while there is an imperative command, best to go
the declarative apply route so that the deployment is in
agreement with the yaml file.
In an update, under the hood, a new replicaset is created with desired number of pods and the old pods are slowly terminated
Undoing a rollout:
kubectl rollout undo deployment/myapp-deployment
The default setting for kubernetes is for a pod to be continually redeployed upon full execution of a job. Explicitly:
apiVersion: v1
kind: Pod
metadata:
name: math-pod
spec:
containers:
- name: math-add
image: ubuntu
command: ['expr', '3', '+', '2']
restartPolicy: Always
This is a little different for deployments. While a replicaset is used to make sure a specified number of pods are running at all times, a job is used to run a set of pods to perform a given task to completion.
Creating a job:
apiVersion: batch/v1
kind: Job
metadata:
name: math-add-job
spec:
completions: 3
template:
spec:
containers:
- name: math-add
image: ubuntu
command: ['expr', '3', '+', '2']
restartPolicy: Never
*** Notice how we used the pod’s spec as a template for the job. In the above case, the job will be run three times.
If there’s a situation where a job may fail one of the completions, it will continue to run until the desired completions have been attained.
Can also make these jobs parallel by including a
parallelism: <num_jobs> line within
spec:
A job that can be scheduled (note how the above job is placed within
jobTemplate:)
apiVersion: batch/v1
kind: CronJob
metadata:
name: reporting-cron-job
spec:
schedule: "*/1 * * * *"
jobTemplate:
spec:
completions: 3
template:
spec:
containers:
- name: math-add
image: ubuntu
command: ['expr', '3', '+', '2']
restartPolicy: Never
Service: K8 object that enables communication between various components between applications via ports
Can connect between applications and users (ex: serving frontend loads to user, connecting to external data sources, etc…)
Frontend to users, frontend/backend pods, external data sources
Loose coupling between microservices between application
DEFAULT Kubernetes service
Gives service inside cluster that other apps inside cluster can access (no external access!)
Helps us group pods together and provide a single interface to access pods in a group
This then allows all pods to be able to scale/create/terminate without impacting communication between the various services
Each service gets an IP and a name (other pods should use this to access)
CAN access via kubernetes proxy by starting it and then navigating to endpoint
kubectl proxy --port=8080
http://localhost:8080/api/v1/proxy/namespaces/<NAMESPACE>/services/<SERVICE-NAME>:<PORT-NAME>/
Useful for debugging services, allowing internal traffic, displaying internal dashboards
Because method requires kubectl as authenticated user, DO NOT EXPOSE TO PUBLIC!
apiVersion: v1
kind: Service
metadata:
name: my-internal-service
spec:
selector:
app: my-app
type: ClusterIP
ports:
- name: http
port: 80
targetPort: 80
protocol: TCP
http://localhost:8080/api/v1/proxy/namespaces/default/services/my-internal-service:80/
Opens a specific port on all the nodes, and any traffic that is sent to this port is forwarded to the service
Need to select a specific pod class to forward this port via
selector; this links the service to the pod class and works
as a built-in loadbalancer
targetPort refers to the port of the pod, whereas
port refers to the port of the service (assumed to be the
same if targetPort not included)
If nodePort is not selected, it will be assigned a value
between 30000-32767
If Node/VM IP address changes, the NodePort must be manually altered
Only useful for purposes of testing/demoing!
apiVersion: v1
kind: Service
metadata:
name: my-nodeport-service
spec:
selector:
app: my-app
type: NodePort
ports:
- name: http
port: 80
targetPort: 80
nodePort: 30036
protocol: TCP
Standard way to expose a service to the internet
Usually implemented via a cloud provider, of which will give a single IP address that will forward all traffic to the service. Usually can’t do this locally! Will need to provision the details there.
Useful for directly exposing a service, meaning you can send almost any kind of traffic to it (HTTP, TCP, UDP, Websockets, etc…)
CAUTION! Each service you expose with a Load Balancer will get its own IP address. This can get expensive quick.
apiVersion: v1
kind: Service
metadata:
name: my-lb-service
labels:
app: my-app
spec:
type: LoadBalancer
ports:
- port: 8080
selector:
app: my-app
Why are the above not enough? While a NodePort / Loadbalancer can get the job done of accessing the specific node,
you will have to implement a proxy-server to redirect from a hard-to-remember nodePort service to the default port 80 (this is where a cloud load balancer would be helpful.. don’t need a proxy!)
Multiple pods providing multiple services will need their OWN load balancers (costly) and then a proxy load balancer to connect them all under the same DNS!
Not to mention enabling SSL so that users can access the website… a nightmare with individual loadbalancers / nodeports and proxy-servers…
Ingress: a way to store all of the above configuration shortfalls into a SINGLE yaml file
Users can access application using a single externally accessible URL
Can be configured to route traffic to different services within the cluster based on URL path
Can easily implement SSL certification
apiVersion: extensions/v1
kind: Deployment
metadata:
name: nginx-ingress-controller
spec:
replicas: 1
selector:
matchLabels:
name: nginx-ingress
template:
metadata:
labels:
name: nginx-ingress
spec:
containers:
- name: nginx-ingress-controller
image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.21.0
args:
- /nginx-configuration-controller
- --configmap=$(POD_NAMESPACE)/nginx-configuration
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath:
metadata.namespace
ports:
- name: http
containerPort: 80
- name: https
containerPort: 443
---
apiVersion: v1
kind: configMap
metadata:
name: nginx-configuration
---
apiVersion: v1
kind: Service
metadata:
name: nginx-ingress
spec:
type: NodePort
ports:
- port: 80
targetPort: 80
protocol: TCP
name: http
- port: 443
targetPort: 443
protocol: TCP
name: https
selector:
name: nginx-ingress
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: nginx-ingress-serviceaccount
Ingress Controller: A deployed reverse proxy/load balancer solution (nginx, HAProxy, Traefik); NOT deployed by default!
Deployment configuration:
within image, we must start the controller service
args: -/nginx-ingress-controller
configMap is useful for parameterizing options that may change within deployment (can begin as blank!)
must pass in pod name and namespace to the environment so that the nginx-service can read the config info within the pod
finally, specify the ports used by ingress controller
Service configuration:
Needed to link the ingress controller to the external world
nginx-ingress links the service to the
deployment
Service Account configuration:
ingress controllers have additional intelligence to monitor K8 cluster and can modify nginx server when something has changed
BUT requires a Service Account with the right set of permissions to interact with the pods
Includes appropriate Roles, ClusterRoles, and RoleBindings
Ingress Resources: Set of rules to configure an ingress
Can route traffic to different applications based on URL (example.com/watch, example.com/wear)
Or even on the domain name itself! (watch.example.com, wear.example.com)
Routed to application services and NOT the pods directly (makes sense)
Create rules for each domain name
Tip: Can route specific urls to their applications, and then create a final rule that routes all other urls to “404 Not Found”
Paths redirect to appropriate endpoint (ex: my-online-store.com/watch)
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: ingress-wear-watch
spec:
rules:
- http:
paths:
- path: /wear
pathType: Prefix
backend:
service:
name: wear-service
port:
number: 80
- path: /watch
pathType: Prefix
backend:
service:
name: watch-service
port:
number: 80
Domain/host names which direct to paths (ex: watch.my-online-store.com)
apiVersion: extensions/v1
kind: Ingress
metadata:
name: ingress-wear-watch
spec:
rules:
- host: wear.my-online-store.com
http:
paths:
- backend:
serviceName: wear-service
servicePort: 80
- host: watch.my-online-store.com
http:
paths:
- backend:
serviceName: watch-service
servicePort: 80
Traffic:
Ingress: network traffic whose source lies in an external network, and sends to the destined node in the private network
Egress: network traffic that begins inside a private network and proceeds through it’s routers to a destination outside of the network
Need rules for each of the flows to each port!
The response does NOT need rules
By default, all pods have an “All Allow” rule so they can interact with each other.
Network Policies: The kubernetes way to restrict communication between pods (ex: in the above, we would set a networkPolicy on the Database Pod such that only ingress traffic from the API Pod on port 3306 is allowed)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: db-policy
namespace: prod
spec:
podSelector:
matchLabels:
role: db
policyTypes:
- Ingress
- Egress
ingress:
- from:
- podSelector:
matchLabels:
name: api-pod
namespaceSelector:
matchLabels:
name: prod
- ipBlock:
cidr: 192.168.5.10/32
ports:
- protocol: TCP
port: 3306
egress:
- to:
- ipBlock:
cidr: 192.168.5.10/32
ports:
- protocol: TCP
- port: 80
Tips to create network policies:
Focus only on the link that you are wanting to enforce (in the above, that would be port 3306 between API and DB)
Begin by specifying the pod which you want to restrict via
podSelector:
Block ALL traffic (Ingress and Egress) via
policyTypes: and then peel away the traffic that should
flow from the perspective of the specified pod
Finally, include the pod/port info that can take Ingress/Egress traffic with respect to the target pod
If you want the policy to only apply a certain namespace, use the
namespaceSelector: argument
If you want to allow a range of IP addresses to hit the pod, use
ipBlock:
The dashes represent OR, while a continuous list represents AND (ex: incoming traffic from pods must have name api-pod AND be in namespace prod OR the traffic must come from cidr 192.68.5.10/32)
Notice the parity between the ingress/egress arguments. In this example, traffic can also flow TO the cidr because of the Egress policy.
To specify multiple policies in one file, just continue to apply
to: / from: conditions below
ingress: / egress:
To persist data across multiple ephemeral pods, volumes are attached to containers and data processed is stored in them
When volume created, can configure in various ways via
volume: (ex: directory on host node)
Use volumeMounts: to mount the volume onto the
container
When pod gets deleted, the data still lives on the host
In the case of multiple nodes, want to use a scalable network storage solution (ex: AWS EBS) since the volume path could have different contents for each node (since they’re different servers).
apiVersion: v1
kind: Pod
metadata:
name: random-number-generator
spec:
containers:
- image: alping
name: alpine
command: ["bin/bash", "-c"]
args: ["shuf -i 0-100 -n 1 >> /opt/number.out;"]
volumeMounts:
- mountPath: /opt
name: data-volume
volumes:
- name: data-volume
hostPath:
path: /data
type: Directory
-OR-
awsElasticBlockStore:
volumeID: <volume-id>
fsType: ext4
Instead of defining a volume within every single pod and having to manually change each time, should go the direction of managing storage more centrally
Strategy: Create a large pool of storage, with users “carving” out pieces as needed
accessModes: come in three flavors: ReadOnlyMany,
ReadWriteOnce, ReadWriteMany
capacity: How much storage would you like to
allocate?
apiVersion: v1
kind: PersistentVolume
metadata:
name: pv-vol1
spec:
accessModes:
- ReadWriteOnce
capacity:
storage: 1Gi
awsElasticBlockStore:
volumeID: <volume-id>
fsType: ext4
persistentVolumeReclaimPolicy: Reclaim
apiVersion: v1
kind: Persistent-Volume-Claim
metadata:
name: myClaim
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage:
500Mi
---
apiVersion: v1
kind: Pod
metadata:
name: mypod
spec:
containers:
...
volumes:
- name: mypd
persistentVolumeClaim:
claimName: myClaim
Persistent Volume Claims: A separate object created by the user to use the persistent volume created by the administrator. Once created, k8 binds the PV to claims based on the request and properties set on the volume
There is a one-to-one relationship between PVs and PVCs, so if a PVC does not utilize all the storage, there’s the remainder can’t be allocated anywhere else
PVCs that can’t find a PV will remain in a pending state until a PV with the proper configuration is created
When a PVC is deleted, the PV is set to retain by default via
persistentVolumeReclaimPolicy: Retain (Retain, Delete,
Recycle-data gets scrubbed)